This invention pertains generally to devices, device interconnection topologies, and methods for communicating data or other information between such devices; more particularly to inter- and intra-device connection and communication topologies and methods for such communication; and most particularly to RAID storage system controllers that increase available storage device interconnect channel capacity by routing controller-to-controller messages to a communication channel separate from the communication channel normally used to communicate the RAID data.
Standard interconnect schemes for sharing memory resources between controllers in respective data storage subsystems are known. Certain controller-to-controller message passing schemes and interconnect topologies are also known. These controllers typically use a message passing scheme to coordinate activities between the controllers. Such message passing schemes utilize the same communication path for passing messages as is used to communicate data between the controllers and the storage devices themselves. Storage devices such as RAID disc drive arrays may conventionally utilize Small Computer Systems Interface (SCSI) protocols over SCSI or Fibre Channel communication channel links. This is a reasonably effective message passing or communication scheme for controller-to-controller messaging when the number of messages is relatively small and the amount of data which is transmitted per message is a small percentage of the available finite communications channel bandwidth.
In one particular conventional RAID controller architecture and communication scheme, such as for example that utilized in certain families of RAID controllers made by Mylex Corporation (a subsidiary of International Business Machines), controller-to-controller messages are communicated through the same communication channels used to communicate data to and from the RAID arrays. These communication channels may include for example, PCI bus, backend disk storage device loops, fiber channel based device loops, or other structures for coupling host and/or storage systems to a storage system controller. While use of the PCI bus, backend disc loops, or other conventional means for such controller-to-controller messaging is not generally a problem or limitation when messages tend to be short, as the majority of such messages tend to be, there are however, problems and/or limitations that may arise where the message is relatively long. These relatively long messages may occur for example for write-back operations, such as Logical Unit (LUN) type I/O operations, where all of the host write data must be mirrored to the partner controller prior to status being returned to the host. As this must be accomplished prior to returning status to the host, it is imperative that it be accomplished as rapidly as possible. The inclusion of the mirrored host write data increases the size of the message as compared message types not containing appreciable data content. Conventionally, this data mirroring is accomplished using the PCI busses, backend disc loops, or the like channels.
In general, these prior art systems are problematic because use of such conventional interconnect schemes utilize finite channel bandwidth that may be better utilized for transferring data to and from the data storage system than for communicating messages between controllers. As a result performance bottlenecks result over the storage system communication channel and/or between respective controller related memory resources. Limitations associated with this disc-loop based scheme as well as with and other conventional controller-to-controller communication schemes may be expected to be more problematic as further demands are made on finite channel bandwidth limitations.
For example, a typical 66 MHz 64-bit PCI bus supports about 532 MB/second. This available bandwidth limits the number of Fibre Channels which can effectively be placed upon the bus without it saturating. Full duplex fiber channel protocol devices operating at 2 Gbit (about 400 MB/sec) may soon become available, and since they will be able to simultaneously send and receive data, their effective bandwidth will double, thereby further compounding the bandwidth limitation issues. Faster PCI busses (or other suitable interconnect bus schemes) may also someday become available, but are not available to meet present or near-term requirements.
Therefore, there exists a need for system, architecture, and method that increase channel available bandwidth and provides the desired device-to-device (or chip-to-chip or component-to-component) and memory access to reduce such performance bottlenecks. More particularly, there exists a need to provide structure, architecture, and method for improving controller-to-controller communication that does not degrade performance of host write I/O operations or other data transfer over the disk channels. There also remains a need for a modular building block architecture which can be used in a controller design to address the above described channel bandwidth and controller messaging limitations. There also remains a need for structure and method for a shared memory controller wherein one controller can directly access memory of its partner controller. There further remains a need for a flexible and expandable out-of-band controller-to-controller interconnection device and method which allows multiple controllers to be connected to provide the desired intercontroller communication while living within the limitations of the available bus architecture constraints, such as the available PCI bus bandwidth limitations. There also remains a need to provide larger sharable data cache memories within the controllers so as to increase the probability that a required data item will be available from a cache within a controller, thereby reducing the number of storage device accesses and the associated impact on system bandwidth.